The Good, the Bad, the Difficult, and the Easy: Something Wrong with Information Retrieval Evaluation?

When we have an idea of student’s preparation (e.g., because of a previous written exam, or a term project, or after having asked the first questions), we even do something more. We ask difficult questions to good students, and we ask easy questions to bad students: ! What’s the point in asking easy questions to good students? They will almost certainly answer correctly, as expected, without providing much information about their preparation. ! What’s the point in asking difficult questions to bad students? They will almost certainly answer wrongly, without providing much information — and incidentally increase examiner’s stress level.